Online Control With Least-Squares Methods

Policy evaluation using least-squares techniques (such as LSTD and iLSTD) have been shown to estimate the value of a policy with far less data than traditional TD techniques. Unfortunately, they make use of policy-dependent statistics that have to be discarded when the policy changes. This makes it difficult to use the techniques for online control problems. In this paper, we explore the effect of policy on the least-squares statistics, distinguishing three fundamental effects. We then introduce the framework of least-squares Sarsa (LSS and iLSS) and empirically evaluate previously suggested approaches for handling data from older policies in the least-squares statistics. We show these approaches can maintain the leastsquares data efficiency in some control problems, identify circumstances where least-squares approaches can be problematic and where special handling of data from older policies improves learning.